{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise - Regularizing a Decision Tree Model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this exercise, you will train a base Decision Tree classification model and investigate the effects of its hyperparameters on the bias/variance trade-off in order to address overfitting/underfitting.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT MODIFY - imports\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Preparation and Set-up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's say we want to try to classify the direction of 1-day price movements of the [E-mini S&P 500](https://www.cmegroup.com/markets/equities/sp/e-mini-sandp500.html) futures contracts. Execute the cell below to load data for approximately 24 years including some technical analysis features inot the `df` DataFrame.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT MODIFY - load data\n",
    "df = pd.read_csv(\"data.csv\")\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT MODIFY - data preview\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be using the following columns for features (`X`) and the `target_cls` column as the classification target (`y`):\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT MODIFY - features and target definition\n",
    "X = df[[\"ATR\", \"ADX\", \"RSI\", \"ClgtEMA10\", \"EMA10gtEMA30\", \"MACDSIGgtMACD\"]]\n",
    "y = df[\"target_cls\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the cell below to split the data 70/30:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT MODIFY - train/test split\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.3, random_state=52, shuffle=False\n",
    ")\n",
    "X_train.shape, X_test.shape, y_train.shape, y_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Baseline Score, \"Base\" Model and Over-/Underfitting Diagnosis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the cell below to find the distibution of the target class in the training set. Note which class (`0` or `1`) is the majority class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT MODIFY - class distribution on the training set\n",
    "print(\"Target class distribution on `y_train`:\")\n",
    "y_train.value_counts() / len(y_train)   # Expected dutput: Target class `1` is the majority class with a 53.36% share"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If instead of training a machine learning model you always predicted the majority class above, what accuracy score would you achieve on the training set?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL IN - What accuracy score would you get if you predicted the majority class (from `y_train`) for all samples in the test set?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above number provides a \"baseline score\" to anchor our efforts to. Now train a basic `DecisionTreeClassifier` with the default hyperparameter values:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT MODIFY - import\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "# FILL IN - Instantiate and train (fit) a DecisionTreeClassifier with the default hyperparameters and random_state=52\n",
    "clf = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now use this base tree model to get predictions on the train/test sets. Run the cells below and compare the test accuracy result to the baseline score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT MODIFY\n",
    "y_pred_train = clf.predict(X_train)\n",
    "y_pred_test = clf.predict(X_test)\n",
    "\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "print(\"Train accuracy:\", accuracy_score(y_train, y_pred_train))\n",
    "print(\"Test accuracy:\", accuracy_score(y_test, y_pred_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given the above train/test accuracy scores, is the base model overfitting, underfitting, or neither?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. The model is overfitting\n",
    "# 2. The model is underfitting\n",
    "# 3. The model is well-fitted\n",
    "# FILL IN - Choose the correct answer\n",
    "answer = 1\n",
    "# The model is clearly severely overfitting as the training accuracy is 100% and the test accuracy is even less than the baseline score."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Regularizing the Decision Tree Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The process of reducing the complexity of a tree-based model, also known as \"pruning\" the model, helps alleviate overfitting. There are two types of pruning:\n",
    "- In the first kind, we prevent the model from reaching too much complexity/density in the first place by adjusting its hyperparameters. This is the method we will use today.\n",
    "- In the second type, known as \"backward pruning\", we allow the unconstrained tree model to reach its maximum complexity (which means it will almost definitely overfit) and then use methods like Cost Complexity Pruning to pare it down. (If interested, see the documentation for the `cost_complexity_pruning_path()` method of the classifier.)\n",
    "\n",
    "Our base `DecisionTreeClassifer` from earlier had no limit imposed on the maximum depth it was allowed to reach. Re-instantiate and re-fit it, this time limiting its maximum depth to 10. Keep `random_state=52`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL IN - Re-train the model with its maximum depth hyperparameter set to 10 and random_state=52\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inspect the new training and test accuracy scores. Is the model doing better than before in terms of over-/under-fitting?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL IN - Print out the train and test accuracy scores - They should be much closer to each other now indicating reduced overfitting\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Absolute performance is not the focus of this exercise; but feel free to compare the test accuracy to our baseline score to see if we were able to eke out a few more basis points of performance with the pruned model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, we could have pruned the tree by increasing the minimum number of samples allowed per leaf node. Re-train and re-evaluate the classifier with the minimum number of samples per leaf set to `100` (from the default value of `1`) to see if this helps reduce variance. **NOTE:** Do not set the maximum depth hyperparameter. We want to inspect the isolated effect of one hyperparameter at a time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL IN - Re-train the model with its minimum samples per leaf hyperparameter set to 100 and random_state=52\n",
    "\n",
    "\n",
    "# FILL IN - Print out the train and test accuracy scores - Again we should see a reduction in overfitting compared to the base model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another major hyperparameter that can help reduce the complexity of the decision tree is the minimum number of samples required to be present before a split is allowed to occur in decision nodes. Change this hyperparameter's value to `500` (from its default value of `2`). As before, inspect its _isolated_ effect on bias/variance by comparing its train/test accuracy score to that of the base decision tree."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL IN - Re-train the model with its minimum samples per split hyperparameter set to 500 and random_state=52\n",
    "\n",
    "\n",
    "# FILL IN - Print out the train and test accuracy scores - Again we should see a reduction in overfitting compared to the base model\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
